Using learning analytics in clinical competency committees: Increasing the impact of competency-based medical education

ABSTRACT Graduate medical education (GME) and Clinical Competency Committees (CCC) have been evolving to monitor trainee progression using competency-based medical education principles and outcomes, though evidence suggests CCCs fall short of this goal. Challenges include that evaluation data are often incomplete, insufficient, poorly aligned with performance, conflicting or of unknown quality, and CCCs struggle to organize, analyze, visualize, and integrate data elements across sources, collection methods, contexts, and time-periods, which makes advancement decisions difficult. Learning analytics have significant potential to improve competence committee decision making, yet their use is not yet commonplace. Learning analytics (LA) is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders. The ACGME released the third edition CCC Guidebook for Programs in 2020 and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs. Principles of LA underpin national GME outcomes data and training across specialties; however, little guidance currently exists on how GME programs can use LA to improve the CCC process. Here we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: 1) Data Quality and Decision Making, and 2) Educator Development.

Learning analytics is the interpretation of multiple data sources gathered on trainees to assess academic progress, predict future performance, and identify potential issues to be addressed with feedback and individualized learning plans (ILPs) [15,17,18]. What distinguishes LA from other educational approaches is systematic data collection and advanced digital interpretation and visualization to inform educational systems [19,20]. These data are necessary to: 1) fully understand educational contexts and guide improvements; 2) advance proficiency among stakeholders to make ethical and accurate summative decisions; and 3) clearly communicate methods, findings, and actionable recommendations for a range of educational stakeholders [15,16,21].
The ACGME released the third edition CCC 'Guidebook' for Programs in 2020 [1] and the 2021 Milestones 2.0 supplement of the Journal of Graduate Medical Education (JGME Supplement) presented important papers that describe evaluation and implementation features of effective CCCs [22]. Principles of LA underpin national GME outcomes data [23] and training across specialties [21,24]; however, our careful review of these documents [1,22] indicate that little guidance currently exists on how GME programs can use LA to improve the CCC process.
In this paper, we outline recommendations for implementing learning analytics for supporting decision making on trainee progress in two areas: (1) Data Quality and Decision-Making and (2) Educator Development.

Data quality and decision making
We outline our long-and short-term recommendations for data quality and decision making in Table 1. In the near future, data quality standards that are actually embedded in information systems would help GME programs ensure that data quality checks occur prior to CCC meetings. In addition, active engagement of residents in the evaluation process [25]; and monitoring of assessor quality are needed [26]. Aspirational goals include implementing systemized data processing to reduce human error [16] and automating ACGME cloud-based data visualization processes to reduce workload redundancies and improve efficiencies.
Accreditation guides should point to strategies that can aid data analysis and decision-making [1,32]. Aspirational goals include collaborating with GME • Add data quality standards to determine targets for data readiness.
• Use process maps and data quality checks to identify systematic missingness and data capture/processing issues.
• Implement data quality reviews prior to CCC meetings to determine which data are ready for decision-making.
• Engage residents in the data review process [25] • Monitor and improve assessor quality [26] • Implement processes to continuously monitor and improve data quality (CQI).
• Implement systematized data processing to reduce human error.
• Advance assessments of reliability and validity.

Data Integration & Synthesis
• Enhance relationships among stakeholders (e.g., CCC members, program faculty, trainees, hospital leadership and data team) to ensure robust data capture • Fully integrate data team as active CCC participants to assist with display, synthesis, and interpretation of multi-source evaluations [27] • Establish processes to determine if intervention is needed at the program or trainee level • Implement data sharing processes with trainees for coproduction of ILPs [28][29][30] • Collaborate with GME data management systems (e.g., New Innovations, MedHub,) to develop guides for management and synthesis of all available data to inform and optimize visualizations and dashboards.
• Include Electronic Health Record Data so patient care processes and outcomes can be considered. 2934

Data Analyses & Visualization
• Identify patterns in the data before interpreting or acting upon it.
• Use accurate visualization tools and consistently match the data to decisions to be made.
• Implement evidence-based strategies for data visualization during CCC meetings [16].
• ACGME should automate cloud-based visualization processes for programs' Milestones data to reduce their data processing burden, and provide program comparisons to aggregated national Milestone data.

Interpretation & Decision-Making
• Establish shared understanding of data interpretation (e.g., standard setting processes, concept mapping) before reviewing individual performance data.
• Focus on data as a whole before focusing on individual trainees.
• Enhance and promote the development of SMART (Specific, Measurable, Achievable, Relevant, Timely) [27,31] goals to inform residents' plans for improvement, including metrics for achievement.
• Better inform GME program directors about Predictive Probability Value (PPV) data to inform CCC decisions and enhance outcomes for trainees and programs [24].
• Capture data interpretations to continuously refine and test decision-making utility.
• Create robust educational databases to benchmark and integrate ACGME PPV data [24] to provide insights about curriculum and rotation structure, trainee trajectory of acquiring competence, and inform readiness for graduation and independent practice decisions [27].

Contributing to Generalizable Knowledge
• Encourage scholarly activities using de-identified data to contribute to the science of CBME.
• Study the informatics of learning as its own sub-discipline.
• Integrate national and local data to provide precise estimates of trainee competency and 'readiness' for competency-based time-variable GME [7,13,14].
• Support networks of Data Teams/Education Scientists and CCC members to share expertise [27]. data management systems (e.g., New Innovations, MedHub) to develop step-by-step protocols for collating and synthesizing all available data to optimize data visualizations and dashboards. For example, programs can currently access the ACGME predictive probability values (PPVs), which indicate the probability of individual trainees reaching Milestone target Level 4 by graduation [21,24]. Figure 1 illustrates the GME LA landscape and calls out existing failures of data integration for CCCs [33]. Promising studies that link educational quality to patient data and outcomes, underscore the importance of data integration and analytics [19,[34][35][36][37]. Integration of patient electronic health record data would complement trainee multisource evaluation data, enhance the CCC decisionmaking process [34,37], and has been shown to enrich resident feedback and learning [38]. Lastly, creating formal networks of data teams, educational scientists, and CCC members to share complementary LA expertise and collaborate on scholarship would further develop this important work [19,23,27].

Educator Development and Learning Analytics
Educators and trainees need to learn how to manage large streams of quantitative [39] and qualitative [40] data output as they flow to customized resident portfolios to inform CCC decisions and Individualized Learning Plans (ILPs) [28,41]. CCC members need a certain level of analytics literacy toward becoming 'Diagnostic Assessors' that address knowledge gaps [27]. Additional opportunities involve having data that informs both faculty and program development [12,42]. Table 2 outlines recommendations for both educator and program development to apply analytics to CCC processes. Rich LA can enhance trainee development and foster a growth mindset promoting self-reflection, self-directed learning, and coproduction of ILPs [1,7,17,25,28,[48][49][50][51].
In terms of educator development, the 2021 JGME supplement [22] proposes that new CCC member orientation includes: 1) assessment and the use of the Milestones; 2) group decisionmaking; 3) awareness of biases; and 4) the impact of CCC decisions on patient care outcomes [12]. We propose that LA and data interpretation training be included in this orientation along with ongoing educator development that promotes a shared mental model of how informatics can support CCC processes and decisions [12,52]. In addition, simultaneously developing educators and trainees could be done by creatively imbedding ongoing innovative learning experiences on data quality and processes into current educational routines [12,53].
In the more distant future, GME would benefit from faculty trained in educational research through formal training and clinical informatics programs sponsored by the National Library of Medicine [44], ACGME [45], and/or biomedical programs for healthcare professionals [46]. This trained cadre could then ensure CCC members are fully informed about practical LA principles. Lastly, more research is needed to improve the utility of LA and effectiveness of CCCs long-term using well-established outcomes (e.g., tracking GME outcomes [23], board certification status, and disciplinary actions) [19,20,54].

Next steps in supporting these considerable efforts
CCCs engage core GME faculty and program coordinators who balance multiple competing demands [55]. Ideally, a well-designed analytics infrastructure allows for the utilization of more data with less effort [32,56]. Investing in additional team members, ideally education scientists, can accelerate use of LA within CCCs [27]. Options for gaining additional resources include: 1) negotiating with the sponsoring institution to obtain funds or share analytic personnel/resources; 2) applying for local, regional, or national funding; 3) using departmental funds; or 4) altering ACGME requirements [57] to require this work.
Mechanisms for funding GME need to change to enable these kinds of innovations. Applying for grants to support this work as educational innovation is possible, but would not lead to a sustained infrastructure to support improvement [58]. The availability of departmental funds will likely vary by discipline, as those with high clinical revenue may be more likely to provide support. Similarly, large programs with research sections or university-based, affiliated or university administered programs may find expertise in informatics departments, schools of public health or education schools to guide integration of learning analytics into GME processes. Altering or adding program roles and full-time equivalent requirements in ACGME Common Program Requirements [57] may be complex due to employment laws, but times are changing, and residency training funding mechanisms also need revisions. A recent communication with Eric Holmboe, MD, Chief Research, Milestones Development and Evaluation Officer at the ACGME, elucidated that the ACGME is beginning a digital transformation and  [34,38].
• Educator development to ensure analysts, GME program leadership, CCC members, and coordinators understand how to interpret and integrate national PPV Milestone data with local program-level multi-source (360-degree) evaluation data [24,43].
• Trainee development to foster a growth mindset, promote reflection & self-directed learning, and fully inform co-production of ILPs.
• Educators work together differently by imbedding ongoing innovative learning experiences into educational work routines.
• More fully develop raters across clinical rotations to improve completion and accuracy. Iteratively check in with CCC members periodically to increase faculty engagement and ownership over assessment/feedback processes.
• Conduct research on how best to improve Learning Analytics and effective CCC processes to enhance competency-based medical education outcomes and competency-based time-variable GME training [13,14] • National or regional networks are needed to study LA and CCC while sharing and pooling data and resources to move programs toward becoming master CBME educators and assessors.

Shared Mental Model of Learning Analytics
• The Shared Mental Model section of the CCC Guidebook 1 p 18 should include an application of data generated by learning analytics and how these data can inform CCC Milestone determination of trainee progression and program improvement [34,37,38].
• Development and refinement of organizational learning models to allow for growth and to take advantage of new technologies.

Educator Participation in Multidisciplinary Teams
• ACGME should require that data teams be trained in data quality, management, and longitudinal processing.
• Programs should link to relevant hospital information systems and engage with learning analytics experts, and national datasets.
• Engage minoritized communities about collection of Diversity, Equity, Inclusion, and Social Justice data [47].
• Establish training programs for physicians, as occurred with biomedical training programs [46], to cover costs of advanced educational science training.
• Fund programs offered through the ACGME and the National Library of Medicine to train Educational Informatics experts [44][45][46] • Data team includes Masters or PhD trained educational scientists or physicians with informatics training with an in-depth understanding of GME program curriculum and competency-based multi-source assessments [40].
exploring learning analytics as part of that transformation (Cite: Personal Communication 1/26/23). In summary, action is needed to fully realize CBME in residency training. Evolutionary pathways for inclusion of LA and educator and trainee development in guiding literature, such as the next CCC Guidebook [1] and ACGME Common Program Requirements [57], would help advance these efforts. Collaborating to coproduce generalizable LA processes with external stakeholders, including Residency Data Management Systems, to inform efficient CCC processes will be essential to advance CBME.

Disclosure statement
No potential conflict of interest was reported by the authors.

Funding
This publication was prepared with financial support from the American Medical Association (AMA). The content reflects the views of the authors and does not purport to reflect the views of AMA or any member of the Accelerating Change in Medical Education Consortium.